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Software  Acquisition  Capability  Maturity 
ModelSM  Pilot  Appraisal  Report,  Version 
1.0 


Abstract:  This  document  summarizes  five  pilot  appraisals  performed  from  the 
third  quarter  of  1995  through  the  first  quarter  of  1996  using  the  Software 
Acquisition  Capability  Maturity  Model  (SA-CMM).  The  pilot  appraisals  used 
Version  0.01  of  the  SA-CMM,  published  in  June  1995;  Version  0.02  of  the  SA- 
CMM,  published  in  February  1996;  and  the  CMM-based  appraisal  methodology  for 
internal  process  improvement  (CBA IPI),  tailored  for  use  with  the  SA-CMM. 


Introduction 


Objectives  This  document  summarizes  five  pilot  appraisals  performed  from  the 

third  quarter  of  1 995  through  the  first  quarter  of  1 996  using  the 
Software  Acquisition  Capability  Maturity  Model  (SA-CMM).  The  pilot 
appraisals  used  Version  0.01  of  the  SA-CMM,  published  in  June  1995; 
Version  0.02  of  the  SA-CMM,  published  in  February  1996;  and  the 
CMM-based  appraisal  methodology  for  internal  process  improvement 
(CBA  IPI),  tailored  for  use  with  the  SA-CMM. 

The  objective  of  the  pilot  appraisals  is  to  gather  data  on  the  validity  of 
the  model  and  the  appraisal  method  when  they  are  applied  to  a  variety  of 
organizations.  The  SA-CMM  author  team  gathered,  captured,  and  used 
data  from  the  pilot  appraisals  to  modify  subsequent  draft  versions  of  the 
model. 

While  the  CBA  IPI  methodology  can  be  used  to  provide  ratings 
[maturity  level  and  key  process  area  (KPA)  goal  satisfaction],  the  SA- 
CMM  pilot  appraisal  findings  included  only  strengths  and  opportunities 
for  improvement  relative  to  practices  in  the  model. 

Each  appraisal  team  included  two  or  more  SA-CMM  authors,  one  CBA 
IPI  lead  assessor,  one  Software  Engineering  Institute  (SEI)  facilitator, 
and  three  or  more  site  team  members. 


Organizations  Five  separate  appraisals  were  performed  for  the  organizations  listed 
appraised  below  between  October  1995  and  March  1996. 

•  Air  Force  Missile  Warning  and  Space  Surveillance  Sensors 

•  Air  Force  Mission  Planning  Systems  Program  Office 

•  Navy  Cruise  Missile  Weapon  Systems  Program  Office 

•  Army  Research,  Development,  and  Engineering  Center  (ARDEC)  Fire 
Control  and  Software  Engineering  Division 

•  Defense  Information  Systems  Agency  (DISA)  Applications  Engineering 
Facility 
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Appraisal 

schedule 


Appraisal 

characteristics 


Pilot  appraisals  were  conducted  over  two  weeks: 

•  Training  (four  days)  -  SA-CMM  (one  day),  CBA IPI  (two  days),  in- 
briefing  and  maturity  questionnaire  (one  day) 

•  Assessment  (five  days)  -  Interviews  and  data  gathering  (three  days), 
draft  findings  preparation  and  briefing  (one  day),  final  findings 
preparation  and  briefing  (one  day) 


The  organizations  appraised  were  selected  to  participate  due  to  the 
differences  in  service,  mission  (life-cycle  phase),  and  domain.  Table  1-1 
summarizes  some  of  the  characteristics  of  the  appraisals  and  of  the 
organizations.  Table  1-1  also  lists  the  members  of  the  appraisal  teams. 


Organization 

Mission 

Domain 

Appraisal 

Team 

Air  Force 
Missile 

Warning  and 
Space 

Surveillance 

Sensors 

Post¬ 
deployment 
software 
support 
(PDSS)  of 
missile  and 
space  sensor 
systems 

five  projects 

Radar  and  optical 
sensor  control 
systems 

•  CBA  lead: 

Larry  Jones 

•  Authors:  Jack 
Cooper,  Jack 
Ferguson 

•  Facilitators: 
Mike  Ginn, 

Jan  Philpot 

•  Site:  Bill  Bahl, 
Pam  Conner, 
Gail  Steele 

Air  Force 
Mission 
Planning 
Systems 
Program  Office 

Development 

of  mission 

planning 

systems  for 

aircraft/ 

electronics/ 

weapons 

combat 

operations 

six  integrated 
product  teams 

Command  and 
control  systems; 
multiple  systems 
integrator,  route 
planning  and 
critical  data 
generator  integral 
to  weapon 
performance 

•  CBA  lead: 
George 

Winters 

•  Authors:  Jack 
Ferguson, 

John 

Marciniak, 

Bob  Webster 

•  Facilitator:  Jan 
Philpot 

•  Site:  John 
Dienes,  Steve 
Dempsey, 

Brian  Gesuale 

Table  1-1.  Appraisal  Characteristics 


continued  on  next  page 
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Appraisal 

characteristics, 

continued 


Organization 

Project 

Domain 

Domain 

Appraisal 

Team 

Navy  Cruise 
Missile 

Weapon 
Systems 
Program  Office 

Development 
and  PDSS  for 
naval 

airbreathing 
cruise  missiles 

three  projects 

Real-time  ship¬ 
board  weapon 
control  systems 

•  CBALead: 
Brenda 

Zettervall 

•  Authors:  Jack 
Ferguson, 

Tony  Guido, 
Jordan 

Matejceck 

•  Facilitator:  Jan 
Philpot 

•  Site:  Jim 
Blackwelder, 
Andrew 

Home,  Marty 
Martinez 

ARDEC  Fire 
Control  and 
Software 
Engineering 
Division 

Development 

and  PDSS  for 

gun-fired 

weapons 

systems, 

including 

Army  mortars 
and  self- 
propelled 
howitzers 

four  projects 

Real-time 
embedded  weapon 
fire  control 
systems 

•  CBALead: 

Jack  Ferguson, 
Larry  Jones 
(co-lead) 

•  Authors:  Jack 
Cooper,  Mike 
Falat,  Matt 
Fisher,  Jack 
Ferguson 

•  Facilitator: 

Scott  Reed 

•  Site:  Laura 
Hojecki, 

Kalman 

Laudon,  Patty 
Lyon,  Joe 
Sierodzinski, 
Don  Stewart 

Table  1-1.  Appraisal  Characteristics  (cont’d) 


continued  on  next  page 
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Appraisal 

characteristics, 

continued 


Organization 

Project 

Domain 

Domain 

Appraisal 

Team 

DISA 

Applications 

Engineering 

Facility 

Development 

and 

maintenance  of 

applications 

for  several 

information 

systems, 

including  the 

global 

command  and 
control  system 

four  projects 

Information 
management  and 
communication 
systems 

•  CBALead: 

Tara  Rumsey 

•  Authors:  Jack 
Cooper,  Mike 
Falat,  Matt 
Fisher,  Jack 
Ferguson 

•  Facilitator: 

Scott  Reed 

•  Site:  Prince 
Billups,  Rosa 
Charity,  Ken 
Kelley,  Denise 
Richardson, 
Nancy 

Summers 

•  Other:  Fred 
Luppino,  Loral 
Federal 

Systems 

Table  1>1.  Appraisal  Characteristics  (cont’d) 
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Overview  of 
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issues 


Planning 

versus 

doing 


Real  and 
apparent 
overlaps 


Added 

explanatory 

material 


In  general,  the  SA-CMM  was  viewed  as  an  accurate  model  of  software 
acquisition.  Site  team  members  and  interview  subjects  reported  that  the 
SA-CMM  describes  “what  they  do  or  should  be  doing  to  acquire 
software-intensive  systems.” 

The  pilot  appraisals  resulted  in  27  comment  forms  issued  to  the  author 
team.  Some  specific  questions  were  raised  about  the  meaning  of  various 
words  in  the  model  and  the  need  for  more  examples  to  explain  some 
activities.  Other  comments  are  detailed  in  the  remainder  of  this  report. 


The  initial  design  of  the  model  appears  to  overemphasize  planning.  Goal 
1  of  every  KPA  in  Version  0.01  requires  that  the  KPA  be  planned,  and 
activities  1  and  2  in  almost  every  KPA  reinforce  the  need  for  plans. 
While  level  2  of  the  model  requires  policies,  plans,  and  procedures,  the 
repetitive  nature  of  these  goals  and  activities  is  not  evident  in  the 
Software  CMM  (SW-CMM)  because  the  SW-CMM  does  not  follow  a 
template. 

Following  the  template,  with  its  emphasis  on  plans,  and  then  following 
the  plans  made  evident  the  repetitive  nature  of  the  writing,  which 
detracted  from  the  active  parts  of  the  model.  Thus  Version  0.02  of  the 
SA-CMM  corrected  this  problem  by  reducing  one  level  of 
documentation:  Instead  of  requiring  policies,  plans,  and  documented 
procedures  for  writing  plans,  SA-CMM  Version  0.02  requires  policies 
and  plans,  but  not  a  documented  procedure  for  writing  plans.  Draft 
templates  for  Version  2  of  the  SW-CMM  also  delete  this  documentation 
requirement. 


The  pilot  appraisals  identified  several  overlaps  and  redundancies  among 
the  project  office  management  (POM),  contract  tracking  and  oversight 
(CT&O),  and  evaluation  and  acceptance  (E&A)  KPAs  of  Version  0.01. 
Some  of  these  overlaps  and  redundancies  were  corrected  in  Version 
0.02,  but  not  all.  Some  apparent  overlaps — e.g.,  risk  identification  in 
POM  with  the  software  acquisition  risk  management  (S ARM)  KPA — 
are  intentional,  since  some  lower  level  activities  lay  the  foundation  for 
higher  level  KPAs.  In  addition,  better  descriptions  of  the  relationships 
between  Level  2  and  Level  3  KPAs  are  needed. 


Some  appraisals  highlighted  the  need  for  a  better  explanation  of  the  use 
of  the  model.  Specifically,  the  introduction  should  emphasize  that 

•  Maintenance  is  viewed  as  “the  acquisition  of  maintenance”  and  is 
thus  subject  to  appropriate  KPAs  just  as  is  the  acquisition  of 
software  development  and  software-related  services. 

•  In-house  developers  and  maintainers  are  considered  as  “contractors” 
by  the  model,  thus  the  model’s  KPAs,  with  appropriate  tailoring, 
apply  to  in-house  development  and  maintenance. 
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of  The  CBA IPI  appraisal  methodology  was  used  for  all  pilot  appraisals, 

team  Team  training  was  modified  from  the  training  for  the  SW-CMM 

appraisals — one  day  was  allotted  for  Introduction  to  the  SA-CMM  and 
two  days  were  allotted  for  CBA  IPI.  The  original  course  contains 
sections  on  interviewing  individuals  and  on  interviewing  groups,  along 
with  associated  data-consolidation  sections.  This  was  reduced  to  one 
section  on  interviewing  techniques  and  one  section  on  data 
consolidation.  These  changes  were  forwarded  to  the  CBA  IPI 
developers  for  possible  inclusion  in  the  course. 


The  pilots  were  scheduled  to  occupy  a  nominal  eight-hour  work  day  as 
opposed  to  the  marathon  sessions  usually  associated  with  a  CMM-based 
appraisal.  Some  of  the  time  was  recouped  in  the  following  ways: 

•  Because  of  the  time  proximity  between  the  team  training  and  the  on¬ 
site  appraisal,  the  kickoff  briefing  was  combined  with  the 
assessment  participants  briefing  and  conducted  during  the  week  of 
team  training. 

•  The  initial  document  review  was  conducted  as  part  of  team  training 
or  during  the  period  between  team  training  and  the  appraisal. 

For  most  of  the  pilots,  this  schedule  did  not  usually  provide  data 
sufficiency  to  rate  all  KPAs,  but,  given  the  lack  of  maturity  of  the 
model,  ratings  were  not  a  goal  of  these  appraisals.  The  schedule  allowed 
for  only  four  or  five  interview  sessions.  To  obtain  deeper  coverage  and 
determine  a  rating  might  require  another  two  or  three  days.  As  it  was, 
KPA  coverage  was  usually  prioritized  based  on  the  SA-CMM  maturity 
model  questionnaire  and  initial  interview  results. 


Since  the  appraisal  training  includes  both  SA-CMM  and  CBA  IPI,  the 
appraisal  team  must  have  some  knowledge  of  maturity  modeling  and  the 
rationale  behind  model-based  process  improvement.  With  knowledgeable 
team  members,  the  training  of  one  day  for  the  SA-CMM  and  two  days  for 
CBA  methodology  were  sufficient.  Future  teams  should  attend  at  least  an 
Introduction  to  the  SA-CMM  course  followed  by  CBA  IPI  or  software 
capability  evaluation  (SCE)  training. 

In  light  of  organization  issues  discussed  in  the  next  section,  site  members 
of  appraisal  teams  must  have  detailed  insight  into  the  organizational 
structure  and  business  practices  of  the  appraised  organization. 

Team  training  included  a  session  in  which  interview  questions  were 
crafted  for  the  target  organization.  This  provided  needed  practice  in 
understanding  the  model’s  application  to  the  organization  and 
understanding  the  organizational  structure.  In  these  appraisals,  the  team 
relied  heavily  on  site  team  members  knowledge  of  the  organization. 
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The  pilot  appraisal  teams  found  that  the  structures  of  acquisition 
organizations  are  significantly  more  diverse  than  those  of  development 
organizations.  Some  organizations  combine  acquisition  with 
development  and  maintenance  activities;  some  are  organized  in 
traditional  project-office  fashion,  while  others  have  multiple  integrated 
product  teams;  some  combine  the  acquisition  of  PDSS  for  legacy 
systems  with  the  acquisition  of  newer  systems;  some  provide  functional 
expertise  to  the  program  management/program  executive  office 
structure,  while  others  are  in  the  direct  chain  of  command. 

There  was  also  variety  in  process-improvement  experience — some 
organizations  have  existing  process  improvement  groups  and  others  do 
not.  Those  organizations  with  improvement  groups  were  generally  able 
to  fold  S  A-CMM  appraisal  results  into  their  ongoing  improvement 
efforts.  Others  decided  to  start  improvement  programs  as  a  result  of  the 
SA-CMM  pilot  appraisals. 

This  diversity  means  that  significant  pre-appraisal  planning  must  be 
done  and  that  more  in-depth  knowledge  of  the  organization  must  be 
imparted  to  the  appraisal  team.  Time  was  added  to  the  training  schedule 
for  training  by  the  site  representatives  and  for  tailoring  the  model  and 
questions  to  the  organization.  In  addition,  organizational  issues  and 
scheduling  complexities  require  at  least  one  week’s  time  between 
training  and  appraisal. 


A  maturity  questionnaire  was  used  immediately  after  the  in-briefing  to 
obtain  data  from  key  managers  on  the  extent  of  institutionalization  of  the 
SA-CMM  KPAs  and  to  help  scope  the  appraisal.  There  were  few 
questions  raised  about  the  questionnaire,  but  each  pilot  site  required  the 
definition  of  the  “project”  and  “acquisition  organization”  terms  as 
applied  to  their  organization. 

Many  written  comments  were  obtained  from  the  questionnaire.  These 
were  collated  by  KPA  and  were  used  in  the  assessment  process  when 
needed. 
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Interview  questions  were  written  by  each  appraisal  team,  with  questions 
from  previous  appraisals  used  for  background.  Although  practices  in  the 
SA-CMM  KPAs  formed  the  basis  for  questions,  each  organization 
required  wording  specific  to  that  organization. 

During  the  Navy  appraisal,  the  team  built  an  interview  question  template 
to  help  guide  the  interview  process.  This  template  helped  teams  organize 
interview  sessions  in  later  appraisals  and  will  be  incorporated  into  team 
training  material. 

All  of  the  interview  sessions  were  several  on  several,  with  project 
leader,  functional  area  representative,  and  manager  groups  tailored  by 
knowledge  gained  of  the  organizational  structure.  One  lesson  learned 
was  that  it  is  best  to  interview  managers  and  project  leads  first  and  last, 
since  they  usually  have  knowledge  about  institutionalization  that  may  be 
lacking  in  the  functional  areas. 

There  was  some  discussion  concerning  the  use  of  directed  versus  open- 
ended  questions.  While  directed  questions  help  fill-in  boxes  in  the  KPA 
wall  charts,  they  can  result  in  yes/no  answers  that  do  not  really  describe 
how  the  organization  works.  Open-ended  questions,  however,  may  not 
yield  answers  to  the  desired  practices  and  can  lead  to  wandering,  time- 
wasting  answers.  They  can  also  result  in  blank  stares,  which  provide 
information  but  can  require  significant  team  time  to  interpret.  In 
addition,  some  answers  apply  to  different  KPAs  and  activities  than 
those  that  generated  the  question.  Tagging  these  answers  to  the  model 
requires  a  significant  depth  of  knowledge  of  the  model.  During  the  pilot 
appraisals,  this  depth  was  provided  by  the  SA-CMM  authors.  Future 
teams  that  do  not  have  author  members  must  undergo  a  more  formal 
Introduction  to  the  SA-CMM  course. 
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Some  Common  Appraisal  Findings 


Apparent 
value  of 
appraisal 
results 


Each  organization  appraised  believed  that  the  results  of  the  appraisal 
were  generally  valid,  accurate,  and  useful  for  input  to  a  self- 
improvement  effort. 


Risk  In  general,  pilot  appraisals  found  a  strong,  proactive  approach  to 

management  software  acquisition  risk  identification,  analysis,  and  mitigation.  While 

some  of  these  activities  were  not  integrated  into  a  formal  program,  most 
projects  were  still  actively  reporting  risks  and  mitigation  plans  at  various 
management  reviews. 
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Appendix  A  Outline  of  Briefing  Format 


Briefing  format 


The  following  is  an  outline  of  the  briefing  format  used: 

The  Assessment 

•  The  SA-CMM  Pilot 

•  The  Assessment  Team 

•  Pilot  Assessment  Objectives 

•  Assessment  Activities 

•  Assessment  Scope 

SA-CMM 

•  SA-CMM  Structure 

•  The  SA-CMM  Key  Process  Areas 

Findings 

•  Findings  Summary 

•  Global  Findings 

•  Software  Acquisition  Planning 

•  Solicitation 

•  Requirements  Development  and  Management 

•  Project  Office  Management 

•  Contract  Tracking  and  Oversight 

•  Evaluation 

•  Transition  and  Support 

•  Level  3  Findings 

•  Non  SA-CMM  Findings 

Next  Steps:  IDEAL^M — An  Integrated  Approach  to  Process 
Improvement 
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